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Abstract. This paper investigates how the peer agent’s learning competency af- 
fects English learners’ reading, engagement, system self-efficacy, and attitudes 
toward the peer agent in a trialogue-based intelligent tutoring system (ITS). Par- 
ticipants learned a summarizing reading strategy in the compare-contrast text 
structure in the ITS. Results detected the significant main effect of the peer 
agent’s learning competency on learners’ performance and on engagement. 
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1 Introduction 


Learning companion agents or peer agents have been included in the intelligent tutor- 
ing systems (ITSs) to facilitate learning [1,2]. The peer agent may provide correct or 
incorrect information, encourage or motivate the human learner, and collaborate or 
compete with the learner [3]. Previous work has shown the adoption of the peer agent 
impacts students’ learning outcomes and system-self efficacy in one-on-one (i.e., peer 
agent and student) learning systems with limited pre-defined dialogue options [1,2]. 
However, relatively little work has investigated the effects of the peer agent’s learning 
competency in trialogue-based learning systems. 

Vygotsky’s [4] zone of proximal development (ZPD) confirms the benefits of co- 
operative learning with peers. When compared to independent learning, students in 
peer learning appear to adopt more effective learning strategies by learning proactive- 
ly in their thinking, questioning, and knowledge sharing [5]. A peer’s learning compe- 
tency can be broadly classified into high, middle and low [1,2]. A high-competency 
peer may tutor other students by providing feedback, hints and correct information, or 
by providing a model of the learning process [1,3]. When student peers have similar 
expertise and competency level, each student takes turns to model their learning and 
provide their perception of the content [5]. However, when there is a mismatch of 
competency and expertise between the student and the peer agent, the low- 
competency peer can enhance the student’s self-efficacy beliefs and increase the stu- 
dent’s self-esteem, confidence, and sense of responsibility [2]. 

Kim et al.’s study focused on a novice instructional design course in a one-on-one 
system. Whether their findings will hold true for other populations, academic skills, or 
in one-on-two trialogue-based learning systems is still unclear. The present study 
attempts to replicate these findings with the English as Foreign Language (EFL) 
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learners as the population, the summarizing strategy as the target academic skill, and a 
trialogue-based intelligent system as the learning setting. We designed two conversa- 
tional computer agents in the ITS, a teacher agent and a peer agent, and investigated 
four research questions: (1) Does the high-competency peer agent improve the EFL 
learners’ reading performance, and (2) enhance the EFL learners’ positive attitude 
toward the peer agent? (3) Does the low-competency peer agent enhance the EFL 
learners’ engagement and (4) the EFL learners’ system self-efficacy? 


2 Method 


Participants consisted of 47 Chinese EFL learners (68% female; Age: M=37, 
SD=8.80), who have studied English for at least 8 years. Each participant was ran- 
domly assigned into one of the three conditions: low-competency, 20% of Jordan’s 
answers were correct (n=18); middle-competency, 50% correct (n=15); and high- 
competency, 80% (n=14) based on gender. Actually, more participants would like to 
participate in the study, but later they rejected because of the requirement for their 
social security number for being paid. This resulted in an uneven distribution in the 
conditions. They first took the pre-survey (30 minutes), then interacted with the 
agents (20 minutes), and finally completed the post-survey (10 minutes). The pre- 
survey consisted of basic demographics questions, self-efficacy survey questions [6] 
(see Table 1), and the Gates MacGinitie reading test [7]. Participants were paid $15 
for this one-hour experiment. 

Participants learned a summarizing strategy in a compare-and-contrast text struc- 
ture. The strategy involves the detection of the signal words that signify similarities 
and differences between two things/persons, and the identification and the justifica- 
tion of the good summary. The summarizing strategy lesson also prompts learners to 
provide a justification of good summaries. The mastery of these skills was evaluated 
by analyzing the quality of the learners’ responses to the related reading comprehen- 
sion questions. The reading comprehension questions were primarily multiple-choice 
questions with three options per question. Only one open-ended question was asked, 
which prompted participants to provide a justification for a good summary. All of 
answers were automatically evaluated by the system. To assess the open-ended re- 
sponse, the semantic match between a learner’s verbal input and the expectations 
were evaluated by Regular-Expressions and latent semantic analysis. See Li, Shubeck 
and Graesser [4] for more details about the automatic scoring and rubrics. 

The self-reported engagement after this reading was accepted as the engagement 
score, with 6-point scale from | (not at all engaged) to 6 (very engaged). The post- 
survey consisted of questions that gauged the students’ system self-efficacy and atti- 
tudes toward the peer agent (see Table 1) with the same scale as prior self-efficacy. 


3 Results and Discussion 


Prior and post system self-efficacy and attitude toward the peer agent used in the 
Principal Components Analysis (PCA) to extract one component, respectively, named 
as prior self-efficacy, system self-efficacy, and attitude (see Table 1). 


The prior self-efficacy component explained 62.93% of the total variance, and its 
regression score was performed on a One-way ANCOVA. Results showed there was 
no difference in participants’ prior self-efficacy, F(2,44)=.194, p>.05, M=1.53, 
SD=.504. The regression score of prior self-efficacy was used as a covariate along 
with pretest reading score in the analyses. The system self-efficacy component ex- 
plained 84.78% variance, and the attitude component, 71.24%. Both regression scores 
were used as dependent variables along with reading score and engagement score. 


Table 1. PCA for Prior Self-efficacy, System Self-efficacy and Attitude toward Peer Agent 


Items in Prior Self-efficacy Loadings _ Community 
If I can't do a job the first time, I keep trying until I can. 838 .703 
Failure just makes me try harder. 837 701 
When I have something unpleasant to do, I stick to it until I finish it. .755 570 
When I make plans, I am certain I can make them work. 737 543 
Eigenvalue 2.520 

Items in System Self-efficacy Loadings Community 
The agents are helpful in learning reading strategies. .964 0.929 
This system could help me improve my reading comprehension. 963 927 
The strategies could help me improve my reading. .903 815 

A conversational discussion is a more valid assessment. 849 .720 
Eigenvalue 3.391 

Items in Attitude Toward Peer Agent Loadings Community 
I enjoy reading with Jordan. .870 0.756 

I like Jordan very much. 841 0.707 

I like Jordan’s reading to be better than me. 821 0.674 
Eigenvalue 2.137 


The ANCOVA on reading scores detected a significant main effect with the control 
of prior self-efficacy, F(2,38)=3.47, p<.05, R’=.241. Results revealed marginal signif- 
icant differences between low (M=8.25, SD=1.406) and high (M=8.58, SD=1.436), 
F(1,38)=4.63, p=.076, and low and middle (M=8.30, SD=1.436) conditions, F(1, 
38)=5.452, p=.075. These findings suggest EFL learners who worked with both high- 
and middle-competency peer agents achieved higher reading performance than those 
with the low-competency agent in the trialogue-based ITS. These findings also sug- 
gest that the high-competency agent facilitates learning not only the novice 
knowledge on the instructional design [2], but also the reading strategies. 

The ANCOVA revealed a marginal significant main effect on engagement with the 
control of prior reading score, F(2,38)=2.540, p=.092, R°=.182. A marginal signifi- 
cant difference existed between low (M=4.86, SD=1.150) and high (M=4.69, 
SD=1.351) groups, F(1,38)=4.96, p=.096. These findings suggest learners who work 
with the low-competency agent tend to be more engaged in learning than those with 
the high-competency agent [2]. The majority of participants achieved extremely low 
pretest scores and consequently may have perceived their reading abilities as “low”. 
This perception may have influenced reported self-efficacy beliefs before the inter- 
vention. When low-achievers work with the low-competency peer agent, they may 
find another person is worse than themselves. With this in mind, their confidence may 


increase, which would improve their overall engagement with the system. However, 
when low-achievers work with a high-competency agent, they may not experience the 
increase in confidence, which may lead to their lower engagement reports. 

Results did not show any significant effects of competency on system self-efficacy 
and attitude toward the peer agent. Our study failed to support the assertion that learn- 
ers who work with the low-competency agent achieve higher self-efficacy [2]. How- 
ever, a trend was found that learners who worked with the low-competency peer agent 
reported higher system self-efficacy (M=.09, SD=1.110), followed by the middle- 
(M=.03, SD=.681) and high competency groups (M=.15, SD=1.179). Similarly, our 
findings failed to support the claim that learners who work with high-competency 
agents have a higher positive attitude towards the peer agent. [3]. However, another 
trend was found that learners who worked with the high-competency peer agent 
(M=.29, SD=1.053) reported they liked the system more than those in the middle 
(M=-.06, SD=.734) and low competency groups (M=—.17, SD=1.145). The insignifi- 
cance effects may be caused either by the small sample size or by the influence of 
other individual differences, such as the experience of using educational techniques. 

In conclusion, peer agents in educational intelligent systems do help to improve 
learning outcomes and enhance student engagement [1,2]. Specifically, the peer agent 
with the high or middle competency facilitates learners to achieve higher academic 
performance, whereas the low-competency agent enhances the learners’ engagement. 
These findings are confirmed both by college students in learning the domain-specific 
academic skills [2], and by EFL learners in learning language reading skills. As both 
high- and low-competency agents benefit learners from different perspectives, the 
peer agent’s competency in the educational ITS should be adaptively designed with 
the consideration of the trade-off of learning outcomes and engagement. 
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